Asymptotic optimal control of multi-class restless bandits

We study the asymptotic optimal control of multi-class restless bandits. A restless bandit is acontrollable process whose state evolution depends on whether or not the bandit is made active. Theaim is to find a control that determines at each decision epoch which bandits to make active in orderto minimize the overall average cost associated to the states the bandits are in. Since finding theoptimal control is typically intractable, we study an asymptotic regime instead that is obtained byletting the number of bandits that can be simultaneously made active grow proportionally with thepopulation of bandits. We consider both a fixed population of bandits as well as a dynamic populationof bandits where bandits can depart and new bandits can arrive over time to the system. We proposea class of priority policies, obtained by solving a linear program, that are proved to be asymptoticallyoptimal under a global attractor property and a technical condition. Indexability of the bandits is notrequired for the result to hold. For a fixed population of bandits, the technical condition reduces tochecking a unichain property. For a dynamic population of bandits we present a large class of restlessbandit problems for which the technical condition is always satisfied. As an example, we present amulti-class M/M/S+M queue, which is inside this class of problems and satisfies the global attractorproperty. Henceforth asymptotic optimality of an index policy follows.In case the bandits are indexable, we prove that Whittle’s index policy is included in the class ofasymptotically optimal policies. This generalizes the result of Weber and Weiss (1990), who showedasymptotic optimality of Whittle’s index policy for a symmetric fixed population of bandits, to thesetting of (i) several classes of bandits, (ii) multiple actions, and (iii) possible arrivals of new bandits.In order to prove the main results we combine fluid-scaling techniques with linear programmingresults. This is a different proof approach than that taken in Weber and Weiss, and, in contrary tothe latter, allows to include arrivals of new bandits to the system.